Eye Tracking Using Eye Odometers

ABSTRACT

Low-power eye tracking for detecting position and movements of a user’s eyes in a head-mounted device (HMD). An eye tracking system for an HMD may include eye tracking cameras and eye odometers that may reduce power consumption of the system. The eye odometers may be used as a low-power component to track relative movement of the user’s eyes with respect to the HMD between frames captured by the eye tracking cameras, which may allow the frame rate of the eye tracking cameras to be reduced.

PRIORITY INFORMATION

This application is a continuation of U.S. Pat. Application Serial No.17/021,912, filed Sep. 15, 2020, which claims benefit of priority ofU.S. Provisional Application Serial No. 62/902,329, filed Sep. 18, 2019,the content of which are hereby incorporated by reference herein intheir entirety.

BACKGROUND

Virtual reality (VR) allows users to experience and/or interact with animmersive artificial environment, such that the user feels as if theywere physically in that environment. For example, virtual realitysystems may display stereoscopic scenes to users in order to create anillusion of depth, and a computer may adjust the scene content inreal-time to provide the illusion of the user moving within the scene.When the user views images through a virtual reality system, the usermay thus feel as if they are moving within the scenes from afirst-person point of view. Similarly, mixed reality (MR) combinescomputer generated information (referred to as virtual content) withreal world images or a real world view to augment, or add content to, auser’s view of the world. The simulated environments of VR and/or themixed environments of MR may thus be utilized to provide an interactiveuser experience for multiple applications, such as applications that addvirtual content to a real-time view of the viewer’s environment,interacting with virtual training environments, gaming, remotelycontrolling drones or other mechanical systems, viewing digital mediacontent, interacting with the Internet, or the like.

An eye tracker is a device for estimating eye positions and eyemovement. Eye tracking systems have been used in research on the visualsystem, in psychology, psycholinguistics, marketing, and as inputdevices for human-computer interaction. In the latter application,typically the intersection of a person’s point of gaze with a desktopmonitor is considered.

SUMMARY

Various embodiments of methods and apparatus for low-power eye trackingin virtual and mixed or augmented reality (VR/AR) applications aredescribed.

Embodiments of methods and apparatus for tracking relative movement of adevice with respect to a user’s head are described in which sensors(referred to herein as head motion sensors or head odometers) are placedat one or more positions in or on the device. In some embodiments of aneye tracking system, to accurately determine the location of the user’seyes with respect to the eye tracking cameras, the controller mayexecute an algorithm that performs a three-dimensional (3D)reconstruction using images captured by the eye tracking cameras togenerate 3D models of the user’s eyes. Signals from the head odometersmay be used to detect movement of the device with respect to the user’seyes. This may allow 3D reconstruction to be performed only whenmovement of the device with respect to the user’s eyes has beendetected, thus significantly reducing power consumption by the eyetracking system. In some embodiments, instead of performing 3Dreconstruction when movement of the device with respect to the user’seyes has been detected, magnitude and direction of the detected motionmay be determined, and 3D models of the user’s eyes previously generatedby the 3D reconstruction method may be adjusted according to themagnitude and direction of the detected motion of the HMD.

Embodiments of methods and apparatus for tracking relative movement ofthe user’s eyes with respect to the HMD using eye odometers are alsodescribed. In some embodiments, sensors (referred to herein as eyemotion sensors or eye odometers) are placed at one or more positions inthe device to augment the eye tracking cameras. The eye odometers may beused as a low-power component to track relative movement of the user’seyes with respect to the device in intervals between the processing offrames captured by the eye tracking cameras at a frame rate.

Embodiments of methods and apparatus for tracking relative movement ofthe user’s eyes with respect to the HMD using low-resolution images arealso described. In some embodiments, the eye tracking cameras themselvesmay capture low-resolution frames, for example by binning pixels on thecamera sensor or by capturing horizontal and vertical stripes or linesof pixels on the camera sensor rather than entire frames. Thislow-resolution information may be used to track relative movement of theuser’s eyes with respect to the device in intervals between theprocessing of full, high-resolution frames captured by the eye trackingcameras.

These eye tracking methods and apparatus may allow the frame rate of theeye tracking cameras to be reduced, for example from 120 frames persecond to 10 frames per second or less, and may also allow 3Dreconstruction to be performed much less often, thus significantlyreducing power consumption by the eye tracking system. The eye trackingmethods may be used alone or in combination in various embodiments.

Embodiments of HMDs that include both head odometers and eye odometers,or alternatively that include both head odometers and eye trackingcameras that capture low-resolution images to track relative movement ofthe eyes, are described. Embodiments of an eye tracking system for anHMD that include both head odometers and eye odometers, or alternativelyboth head odometers and eye tracking cameras that capture low-resolutionimages to track relative movement of the eyes, may, for example, furtherreduce the frequency at which 3D reconstruction is performed, and mayalso reduce the frequency at which two-dimensional (2D) image processingof frames captured by the eye tracking cameras is performed to furtherreduce power consumption of the eye tracking system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example VR/AR HMD that implements an eye trackingsystem, according to some embodiments.

FIG. 2 illustrates an example VR/AR HMD that implements an eye trackingsystem that includes sensors to detect movement of the HMD with respectto the user’s eyes, according to some embodiments.

FIG. 3 is a flowchart of an eye tracking method in which sensors areused to detect movement of the HMD with respect to the user’s eyes andin which 3D reconstruction is performed only when movement of the HMD isdetected, according to some embodiments.

FIG. 4 is a flowchart of an eye tracking method in which sensors areused to detect movement of the HMD with respect to the user’s eyes andin which a 3D model of the eye is adjusted when movement of the HMD isdetected, according to some embodiments.

FIG. 5 illustrates an example VR/AR HMD that implements an eye trackingsystem that includes sensors that are used to track movement of the eyesin intervals between the processing of frames captured by the eyetracking cameras, according to some embodiments.

FIG. 6 is a flowchart of an eye tracking method in which sensors areused to track movement of the eyes in intervals between the processingof frames captured by the eye tracking cameras, according to someembodiments.

FIG. 7 illustrates an example VR/AR HMD that implements an eye trackingsystem in which low-resolution frames are captured by the eye trackingcameras and used to track movement of the eyes in intervals between theprocessing of high-resolution frames captured by the eye trackingcameras, according to some embodiments.

FIGS. 8A and 8B illustrate example low-resolution frames captured by theeye tracking cameras, according to some embodiments.

FIG. 9 is a flowchart of an eye tracking method in which low-resolutionframes are captured by the eye tracking cameras and used to trackmovement of the eyes in intervals between the processing ofhigh-resolution frames captured by the eye tracking cameras, accordingto some embodiments.

FIG. 10 illustrates an example VR/AR HMD that implements an eye trackingsystem that includes head odometers that detect movement of the HMD withrespect to the user’s eyes and eye odometers that track movement of theeyes in intervals between the processing of frames captured by the eyetracking cameras, according to some embodiments.

FIG. 11 is a flowchart of an eye tracking method in which head odometersare used detect movement of the HMD with respect to the user’s eyes andeye odometers are used to track movement of the eyes in intervalsbetween the processing of frames captured by the eye tracking cameras,according to some embodiments.

FIG. 12 is a block diagram illustrating an example VR/AR system thatincludes components of an eye tracking system as illustrated in FIGS. 2through 11 , according to some embodiments.

FIG. 13 illustrates an alternative VR/AR device that includes an eyetracking system with head odometers that detect movement of the devicewith respect to the user’s eyes and eye odometers that track movement ofthe eyes in intervals between the processing of frames captured by theeye tracking cameras, according to some embodiments.

This specification includes references to “one embodiment” or “anembodiment.” The appearances of the phrases “in one embodiment” or “inan embodiment” do not necessarily refer to the same embodiment.Particular features, structures, or characteristics may be combined inany suitable manner consistent with this disclosure.

“Comprising.” This term is open-ended. As used in the claims, this termdoes not foreclose additional structure or steps. Consider a claim thatrecites: “An apparatus comprising one or more processor units ....” Sucha claim does not foreclose the apparatus from including additionalcomponents (e.g., a network interface unit, graphics circuitry, etc.).

“Configured To.” Various units, circuits, or other components may bedescribed or claimed as “configured to” perform a task or tasks. In suchcontexts, “configured to” is used to connote structure by indicatingthat the units/circuits/components include structure (e.g., circuitry)that performs those task or tasks during operation. As such, theunit/circuit/component can be said to be configured to perform the taskeven when the specified unit/circuit/component is not currentlyoperational (e.g., is not on). The units/circuits/components used withthe “configured to” language include hardware - for example, circuits,memory storing program instructions executable to implement theoperation, etc. Reciting that a unit/circuit/component is “configuredto” perform one or more tasks is expressly intended not to invoke 35U.S.C. § 112, paragraph (f), for that unit/circuit/component.Additionally, “configured to” can include generic structure (e.g.,generic circuitry) that is manipulated by software or firmware (e.g., anFPGA or a general-purpose processor executing software) to operate inmanner that is capable of performing the task(s) at issue. “Configureto” may also include adapting a manufacturing process (e.g., asemiconductor fabrication facility) to fabricate devices (e.g.,integrated circuits) that are adapted to implement or perform one ormore tasks.

“First,” “Second,” etc. As used herein, these terms are used as labelsfor nouns that they precede, and do not imply any type of ordering(e.g., spatial, temporal, logical, etc.). For example, a buffer circuitmay be described herein as performing write operations for “first” and“second” values. The terms “first” and “second” do not necessarily implythat the first value must be written before the second value.

“Based On” or “Dependent On.” As used herein, these terms are used todescribe one or more factors that affect a determination. These terms donot foreclose additional factors that may affect a determination. Thatis, a determination may be solely based on those factors or based, atleast in part, on those factors. Consider the phrase “determine A basedon B.” While in this case, B is a factor that affects the determinationof A, such a phrase does not foreclose the determination of A from alsobeing based on C. In other instances, A may be determined based solelyon B.

“Or.” When used in the claims, the term “or” is used as an inclusive orand not as an exclusive or. For example, the phrase “at least one of x,y, or z” means any one of x, y, and z, as well as any combinationthereof.

DETAILED DESCRIPTION

Various embodiments of methods and apparatus for eye tracking in virtualand mixed or augmented reality (VR/AR) applications are described. AVR/AR system may include a device such as a headset, helmet, goggles, orglasses (referred to herein as a head-mounted device (HMD)) thatincludes a display (e.g., left and right displays) for displaying framesincluding left and right images in front of a user’s eyes to thusprovide three-dimensional (3D) virtual views to the user. A VR/AR systemmay also include a controller. The controller may be implemented in theHMD, or alternatively may be implemented at least in part by an externaldevice (e.g., a computing system) that is communicatively coupled to theHMD via a wired or wireless interface. The controller may include one ormore of various types of processors, image signal processors (ISPs),graphics processing units (GPUs), coder/decoders (codecs), and/or othercomponents for processing and rendering video and/or images. Thecontroller may render frames (each frame including a left and rightimage) that include virtual content based at least in part on the inputsobtained from cameras and other sensors on the HMD, and may provide theframes to a projection system of the HMD for display.

The VR/AR system may include an eye tracking system (which may also bereferred to as a gaze tracking system). Embodiments of an eye trackingsystem for VR/AR systems are described that include at least one eyetracking camera (e.g., infrared (IR) cameras) positioned at each side ofthe user’s face and configured to capture images of the user’s eyes. Theeye tracking system may also include a light source (e.g., an IR lightsource) that emits light (e.g., IR light) towards the user’s eyes. Aportion of the IR light is reflected off the user’s eyes to the eyetracking cameras. The eye tracking cameras, for example located at ornear edges of the HMD display panel(s), capture images of the user’seyes from the IR light reflected off the eyes. Images captured by theeye tracking system may be analyzed by the controller to detect features(e.g., pupil), position, and movement of the user’s eyes, and/or todetect other information about the eyes such as pupil dilation. Forexample, the point of gaze on the display estimated from the eyetracking images may enable gaze-based interaction with content shown onthe near-eye display of the HMD. Other applications may include, but arenot limited to, creation of eye image animations used for avatars in aVR/AR environment.

An HMD may be a mobile device with an internal source of power. Thus,reducing power consumption to increase the life of the power source is aconcern. Eye tracking systems (both the camera and processingcomponents) consume power. Embodiments of methods and apparatus forproviding low-power eye tracking systems are described that may reducethe amount of power consumed by the eye tracking hardware and softwarecomponents of an HMD.

A key to providing accurate eye tracking is knowing the location of theuser’s eyes with respect to the eye tracking cameras. In someembodiments of an eye tracking system, to accurately determine thelocation of the user’s eyes with respect to the eye tracking cameras,the controller may execute an algorithm that performs athree-dimensional (3D) reconstruction using images captured by the eyetracking cameras to generate 3D models of the user’s eyes. The 3D modelsof the eyes indicate the 3D position of the eyes with respect to the eyetracking cameras, which allows the eye tracking algorithms executed bythe controller to accurately track eye movement.

A key element in accurate eye tracking is robustness in regards todevice movement on the user’s head. The HMD may move on the user’s headduring use. In addition, the user may remove the device and put it backon. In either case, an initial calibration of the eye tracking systemperformed using 3D reconstruction may be invalidated. However, there isan inherent ambiguity in an eye tracking system between whether theuser’s eyes move with respect to the cameras/HMD or whether thecameras/HMD move with respect to the user’s eyes. A solution to thisambiguity is to perform the 3D reconstruction for every frame capturedby the eye tracking cameras. However, 3D reconstruction is expensivecomputationally, and consumes a significant amount of power.

Embodiments of methods and apparatus for tracking relative movement of adevice with respect to a user’s head are described in which sensors(referred to herein as head motion sensors or head odometers) are placedat one or more positions in or on the device, for example at or near theuser’s ears to primarily track pitch and at or near the bridge of thenose to primarily track y movement. Signals from the head odometers maybe used to detect movement of the device with respect to the user’seyes. This may allow 3D reconstruction to be performed only whenmovement of the device with respect to the user’s eyes has beendetected, thus significantly reducing power consumption by the eyetracking system. When no movement of the device is detected, 2D imageprocessing of frames captured by the eye tracking cameras (which is muchless expensive computationally than 3D image processing) may beperformed to track the user’s eyes. In some embodiments, a 3Dreconstruction may be performed periodically (e.g., once a second, oronce every N frames) to prevent error/drift accumulation even ifmovement of the device has not been detected.

In some embodiments, instead of performing 3D reconstruction whenmovement of the device with respect to the user’s eyes has beendetected, magnitude and direction of the detected motion may bedetermined, and 3D models of the user’s eyes previously generated by the3D reconstruction method may be adjusted according to the magnitude anddirection of the detected motion of the HMD. In these embodiments, a 3Dreconstruction may be performed periodically (e.g., once a second, oronce every N frames) to prevent error/drift accumulation.

In addition, embodiments of methods and apparatus for reducing powerconsumption of an eye tracking system are described in which sensors(e.g., photosensors or photodiodes, referred to herein as eye motionsensors or eye odometers) are placed at one or more positions in thedevice to augment the eye tracking cameras. The eye odometers may beused as a low-power component to track relative movement of the user’seyes with respect to the device in intervals between the processing offrames captured by the eye tracking cameras. The 3D models generatedfrom the images captured by the eye tracking cameras provide absolutegaze information, while the data captured by the eye odometers inintervals between the processing of the camera frames is processed toprovide a relative update to the previously known and trusted 3D models.This may allow the frame rate of the eye tracking cameras to be reduced,for example from 120 frames per second to 10 frames per second or less,and may also allow 3D reconstruction to be performed much less often,thus significantly reducing power consumption by the eye trackingsystem. Reducing the frame rate of the eye tracking cameras byaugmenting eye tracking with the eye odometers may also significantlyreduce bandwidth usage and latency of the eye tracking system.

In addition, embodiments of methods and apparatus for reducing powerconsumption of an eye tracking system are described in which the eyetracking cameras may capture low-resolution frames, for example bybinning pixels on the camera sensor or by capturing horizontal andvertical stripes or lines of pixels on the camera sensor rather thanentire frames (high-resolution frames). This low-resolution informationmay be used to track relative movement of the user’s eyes with respectto the device in intervals between the processing of full,high-resolution frames captured by the eye tracking cameras. This mayallow 3D reconstruction to be performed much less often, thussignificantly reducing power consumption by the eye tracking system. Inthese embodiments, the eye tracking cameras themselves may be viewed as“eye odometers” when capturing and processing the low-resolution frames.A high-resolution frame captured by a camera sensor contains more pixelsthan a low-resolution frame captured by the same camera sensor. Ahigh-resolution frame may include most or all pixels that the camerasensor is capable of capturing, while a low-resolution frame may includesignificantly fewer pixels than the same camera sensor is capable ofcapturing. In some embodiments, a camera sensor may be switched, forexample by a controller comprising one or more processors, from ahigh-resolution mode in which high-resolution frames are captured, to alow-resolution mode in which low-resolution frames are captured, forexample by binning pixels on the camera sensor or by capturinghorizontal and vertical stripes or lines of pixels on the camera sensor.

The head odometer methods and eye odometer methods described above maybe used alone or in combination. Embodiments of HMDs that include headodometers to reduce power consumption of the eye tracking system aredescribed. Embodiments of HMDs that include eye odometers (eitherimplemented by sensors or by the eye tracking cameras) to reduce powerconsumption and bandwidth usage of the eye tracking system are alsodescribed. In addition, embodiments of HMDs that include both headodometers and eye odometers are described. Embodiments of an eyetracking system for an HMD that include both head odometers and eyeodometers may, for example, further reduce the frequency at which 3Dreconstruction is performed, and may also reduce the frequency at which2D image processing of frames captured by the eye tracking cameras isperformed to further reduce power consumption of the eye trackingsystem.

While embodiments of an eye tracking system are generally describedherein as including at least one eye tracking camera positioned at eachside of the user’s face to track the gaze of both of the user’s eyes, aneye tracking system may also be implemented that includes at least oneeye tracking camera positioned at only one side of the user’s face totrack the gaze of only one of the user’s eyes.

FIG. 1 shows a side view of an example VR/AR HMD 100 that implements aneye tracking system, according to some embodiments. Note that HMD 100 asillustrated in FIG. 1 is given by way of example, and is not intended tobe limiting. In various embodiments, the shape, size, and other featuresof an HMD 100 may differ, and the locations, numbers, types, and otherfeatures of the components of an HMD 100 may vary. VR/AR HMD 100 mayinclude, but is not limited to, a display 110 and two optical lenses(eyepieces) 120, mounted in a wearable housing or frame. As shown inFIG. 1 , HMD 100 may be positioned on the user 190′s head such that thedisplay 110 and eyepieces 120 are disposed in front of the user’s eyes192. The user looks through the eyepieces 120 onto the display 110.

A controller 160 for the VR/AR system may be implemented in the HMD 100,or alternatively may be implemented at least in part by an externaldevice (e.g., a computing system) that is communicatively coupled to HMD100 via a wired or wireless interface. Controller 160 may include one ormore of various types of processors, image signal processors (ISPs),graphics processing units (GPUs), coder/decoders (codecs), and/or othercomponents for processing and rendering video and/or images. Controller160 may render frames (each frame including a left and right image) thatinclude virtual content based at least in part on the inputs obtainedfrom the sensors, and may provide the frames to a projection system ofthe HMD 100 for display to display 110. FIG. 12 further illustratescomponents of an HMD and VR/AR system, according to some embodiments.

The eye tracking system may include, but is not limited to, one or moreeye tracking cameras 140 and an IR light source 130. IR light source 130(e.g., IR LEDs) may be positioned in the HMD 100 (e.g., around theeyepieces 120, or elsewhere in the HMD 100) to illuminate the user’seyes 192 with IR light. At least one eye tracking camera 140 (e.g., anIR camera, for example a 400×400 pixel count camera or a 600×600 pixelcount camera, that operates at 850 nm or 940 nm, or at some other IRwavelength, and that captures frames at a rate of 60-120 frames persecond (FPS)) is located at each side of the user 190′s face. In variousembodiments, the eye tracking cameras 140 may be positioned in the HMD100 on each side of the user 190′s face to provide a direct view of theeyes 192, a view of the eyes 192 through the eyepieces 120, or a view ofthe eyes 192 via reflection off hot mirrors or other reflectivecomponents. Note that the location and angle of eye tracking camera 140is given by way of example, and is not intended to be limiting. WhileFIG. 1 shows a single eye tracking camera 140 located on each side ofthe user 190′s face, in some embodiments there may be two or more eyetracking cameras 140 on each side of the user 190′s face.

A portion of IR light emitted by light source(s) 130 reflects off theuser 190′s eyes and is captured by the eye tracking cameras 140 to imagethe user’s eyes 192. Images captured by the eye tracking cameras 140 maybe analyzed by controller 160 to detect features (e.g., pupil),position, and movement of the user’s eyes 192, and/or to detect otherinformation about the eyes 192 such as pupil dilation. For example, thepoint of gaze on the display 110 may be estimated from the eye trackingimages to enable gaze-based interaction with content shown on thedisplay 110. As another example, in some embodiments, the informationcollected by the eye tracking system may be used to adjust the renderingof images to be projected, and/or to adjust the projection of the imagesby the projection system of the HMD 100, based on the direction andangle at which the user 190′s eyes are looking.

Embodiments of an HMD 100 with an eye tracking system as illustrated inFIG. 1 may, for example, be used in augmented or mixed (AR) applicationsto provide augmented or mixed reality views to the user 190. While notshown, in some embodiments, HMD 100 may include one or more sensors, forexample located on external surfaces of the HMD 100, that collectinformation about the user 190′s external environment (video, depthinformation, lighting information, etc.); the sensors may provide thecollected information to controller 160 of the VR/AR system. In someembodiments, the sensors may include one or more visible light cameras(e.g., RGB video cameras) that capture video of the user’s environmentthat may be used to provide the user 190 with a virtual view of theirreal environment. In some embodiments, video streams of the realenvironment captured by the visible light cameras may be processed bythe controller of the HMD 100 to render augmented or mixed realityframes that include virtual content overlaid on the view of the realenvironment, and the rendered frames may be provided to the projectionsystem of the HMD 100 for display on display 110. In some embodiments,the display 110 emits light in the visible light range and does not emitlight in the IR range, and thus does not introduce noise in the eyetracking system. Embodiments of the HMD 100 with an eye tracking systemas illustrated in FIG. 1 may also be used in virtual reality (VR)applications to provide VR views to the user 190. In these embodiments,the controller of the HMD 100 may render or obtain virtual reality (VR)frames that include virtual content, and the rendered frames may beprovided to the projection system of the HMD 100 for display on display110.

A key to providing accurate eye tracking is knowing the location of theuser’s eyes 192 with respect to the eye tracking cameras 140. In someembodiments of an eye tracking system, to accurately determine thelocation of the user’s eyes with respect to the eye tracking cameras,the controller 160 may perform a 3D reconstruction using images capturedby the eye tracking cameras 140 to generate 3D models of the user’s eyes192. The 3D models of the eyes 192 indicate the 3D position of the eyes192 with respect to the eye tracking cameras 140 which allows the eyetracking algorithms executed by the controller 160 to accurately trackeye movement. However, a key element in accurate eye tracking isrobustness in regards to device movement on the user’s head. An initialcalibration performed using 3D reconstruction may be invalidated bymovement or removal of the HMD 190. However, there is an inherentambiguity in an eye tracking system between whether the user’s eyes 192move with respect to the cameras 140 or whether the cameras 140 movewith respect to the user’s eyes 192. A conventional solution to thisambiguity is to perform the 3D reconstruction for every frame capturedby the eye tracking cameras 140. However, 3D reconstruction is expensivecomputationally, and consumes a significant amount of power.

FIG. 2 illustrates an example VR/AR HMD 200 that implements an eyetracking system that includes sensors (referred to as head motionsensors or head odometers) to detect movement of the HMD 200 withrespect to the user’s eyes 292, according to some embodiments. Note thatHMD 200 as illustrated in FIG. 2 is given by way of example, and is notintended to be limiting. In various embodiments, the shape, size, andother features of an HMD 200 may differ, and the locations, numbers,types, and other features of the components of an HMD 200 may vary.VR/AR HMD 200 may include, but is not limited to, a display 210 and twoeyepieces 220, mounted in a wearable housing or frame. A controller 260for the VR/AR system may be implemented in the HMD 200, or alternativelymay be implemented at least in part by an external device that iscommunicatively coupled to HMD 200 via a wired or wireless interface.Controller 260 may include one or more of various types of processors,image signal processors (ISPs), graphics processing units (GPUs),coder/decoders (codecs), and/or other components for processing andrendering video and/or images. FIG. 12 further illustrates components ofan HMD and VR/AR system, according to some embodiments.

The HMD 200 may include an eye tracking system that includes, but is notlimited to, one or more eye tracking cameras 240 and an IR light source230. IR light source 230 (e.g., IR LEDs) may be positioned in the HMD200 (e.g., around the eyepieces 220, or elsewhere in the HMD 200) toilluminate the user’s eyes 292 with IR light. At least one eye trackingcamera 240 (e.g., an IR camera, for example a 400×400 pixel count cameraor a 600×600 pixel count camera, that operates at 850 nm or 940 nm, orat some other IR wavelength, and that captures frames at a rate of60-120 frames per second (FPS)) is located at each side of the user290′s face. In various embodiments, the eye tracking cameras 240 may bepositioned in the HMD 200 on each side of the user 290′s face to providea direct view of the eyes 292, a view of the eyes 292 through theeyepieces 220, or a view of the eyes 292 via reflection off hot mirrorsor other reflective components. Note that the location and angle of eyetracking camera 240 is given by way of example, and is not intended tobe limiting. While FIG. 2 shows a single eye tracking camera 240 locatedon each side of the user 290′s face, in some embodiments there may betwo or more eye tracking cameras 240 on each side of the user 290′sface.

To track relative movement of the HMD 200 with respect to the user’seyes, the eye tracking system may also include sensors 242 (referred toherein as head motion sensors or head odometers) placed at one or morepositions in or on the device, for example at or near the user’s ears(242A) to primarily track pitch and at or near the bridge of the nose(242B) to primarily track y movement. Signals from the head odometers242A and 242B may be used to detect movement of the HMD 200 with respectto the user’s eyes. This may allow 3D reconstruction to be performedonly when movement of the HMD 200 with respect to the user’s eyes hasbeen detected, thus significantly reducing power consumption by the eyetracking system. When no movement of the HMD 200 is detected, 2D imageprocessing of frames captured by the eye tracking cameras 240 (which ismuch less expensive computationally than 3D image processing) may beperformed to track the user’s eyes 292. In some embodiments, a 3Dreconstruction may be performed periodically (e.g., once a second, oronce every N frames) to prevent error/drift accumulation even ifmovement of the HMD 200 has not been detected.

In some embodiments, instead of performing 3D reconstruction whenmovement of the HMD 200 with respect to the user’s eyes 292 has beendetected, magnitude and direction of the detected motion may bedetermined, and 3D models of the user’s eyes 292 previously generated bythe 3D reconstruction method may be adjusted according to the magnitudeand direction of the detected motion of the HMD 200. In theseembodiments, a 3D reconstruction may be performed periodically (e.g.,once a second, or once every N frames) to prevent error/driftaccumulation.

Examples of different technologies that may be used to implement headodometers 242 in an eye tracking system are given below in the sectiontitled Example sensors.

FIG. 3 is a flowchart of an eye tracking method in which sensors areused to detect movement of the HMD with respect to the user’s eyes andin which 3D reconstruction is performed only when movement of the HMD isdetected, according to some embodiments. The method of FIG. 3 may, forexample, be performed in a VR/AR system as illustrated in FIG. 2 .

As indicated at 310, a 3D reconstruction method may be performed togenerate 3D models of the user’s eyes using frame(s) captured by the eyetracking cameras. For example, the 3D reconstruction may be performed atinitialization/calibration of the HMD. The 3D models of the eyesindicate the 3D position of the eyes with respect to the eye trackingcameras, which allows the eye tracking algorithms executed by thecontroller to accurately track eye movement with respect to the HMD.

As indicated at 320, 2D image processing of frames captured by the eyetracking cameras may be performed to track movement of the user’s eyeswith respect to the HMD. The eye movement tracked by the 2D imageprocessing may, for example, be used to determine the point of gaze onthe display of the HMD.

As indicated at 330, relative movement of the HMD with respect to theuser’s head may be tracked using sensors on the HMD. A key element inaccurate eye tracking is robustness in regards to device movement on theuser’s head. If the HMD moves on the user’s head during use, the initialcalibration of the eye tracking system performed at element 310 using 3Dreconstruction may be invalidated. However, there is an inherentambiguity in an eye tracking system between whether the user’s eyes movewith respect to the cameras/HMD or whether the cameras/HMD move withrespect to the user’s eyes. To track movement of the HMD with respect tothe user’s eyes, sensors (referred to as head motion sensors or headodometers) are placed at one or more positions in or on the HMD, forexample as illustrated in FIG. 2 . Signals from the head odometers maybe processed by the controller to track movement of the HMD with respectto the user’s eyes.

At 340, as long as the controller does not detect movement of the HMDwith respect to the user’s eyes from the head odometer signals, the eyetracking method may continue to perform 2D image processing to trackmovement of the user’s eyes with respect to the HMD as indicated at 320.At 340, if the controller detects movement of the HMD with respect tothe user’s eyes, the method returns to element 310 to again perform the3D reconstruction method to generate new 3D models of the user’s eyesusing frame(s) captured by the eye tracking cameras, which allows theeye tracking algorithms executed by the controller to continue toaccurately track eye movement with respect to the HMD.

The method of FIG. 3 may allow 3D reconstruction to be performed onlywhen movement of the HMD with respect to the user’s eyes has beendetected, thus significantly reducing power consumption by the eyetracking system. When no movement of the HMD is detected, 2D imageprocessing of frames captured by the eye tracking cameras (which is muchless expensive computationally than 3D image processing) may beperformed to track the user’s eyes. In some embodiments, a 3Dreconstruction may be performed periodically (e.g., once a second, oronce every N frames) to prevent error/drift accumulation even ifmovement of the HMD with respect to the user’s eyes has not beendetected.

FIG. 4 is a flowchart of an eye tracking method in which sensors areused to detect movement of the HMD with respect to the user’s eyes andin which a 3D model of the eye is adjusted when movement of the HMD isdetected, according to some embodiments. The method of FIG. 4 may, forexample, be performed in a VR/AR system as illustrated in FIG. 2 .

As indicated at 420, 2D image processing of frames captured by the eyetracking cameras may be performed to track movement of the user’s eyeswith respect to the HMD. The eye movement tracked by the 2D imageprocessing may, for example, be used to determine the point of gaze onthe display of the HMD.

As indicated at 430, relative movement of the HMD with respect to theuser’s head may be tracked using sensors on the HMD. To track movementof the HMD with respect to the user’s eyes, sensors (referred to as headmotion sensors or head odometers) are placed at one or more positions inor on the HMD, for example as illustrated in FIG. 2 . Signals from thehead odometers may be processed by the controller to track movement ofthe HMD with respect to the user’s eyes.

At 440, as long as the controller does not detect movement of the HMDwith respect to the user’s eyes from the head odometer signals, the eyetracking method may continue to perform 2D image processing to trackmovement of the user’s eyes with respect to the HMD as indicated at 420.At 440, if the controller detects movement of the HMD with respect tothe user’s eyes, then as indicated at 450 the 3D models generated at 410may be adjusted by the controller, and the method may continue toperform 2D image processing to track movement of the user’s eyes withrespect to the HMD as indicated at 420. Magnitude and direction of themovement may be determined from the head odometer signals. Thisinformation may then be used by the controller to adjust the 3D models,which allows the eye tracking algorithms executed by the controller tocontinue to accurately track eye movement with respect to the HMD using2D image processing without requiring expensive 3D reconstruction to beperformed. However, in some embodiments, a 3D reconstruction may beperformed periodically (e.g., once a second, or once every N frames) torecalibrate the eye tracking system.

The method of FIG. 4 may allow 3D reconstruction to be performed lessfrequently than the method of FIG. 3 , thus further reducing powerconsumption by the eye tracking system. When no movement of the HMD isdetected, 2D image processing of frames captured by the eye trackingcameras (which is much less expensive computationally than 3D imageprocessing) may be performed to track the user’s eyes. When movement isdetected, the 3D models can be adjusted without requiring an expensive3D reconstruction. A 3D reconstruction may need to be performed onlyoccasionally to prevent error/drift accumulation.

FIG. 5 illustrates an example VR/AR HMD that implements an eye trackingsystem that includes sensors that are used to track movement of the eyesin intervals between the processing of frames captured by the eyetracking cameras, according to some embodiments. Note that HMD 500 asillustrated in FIG. 5 is given by way of example, and is not intended tobe limiting. In various embodiments, the shape, size, and other featuresof an HMD 500 may differ, and the locations, numbers, types, and otherfeatures of the components of an HMD 500 may vary. VR/AR HMD 500 mayinclude, but is not limited to, a display 510 and two eyepieces 520,mounted in a wearable housing or frame. A controller 560 for the VR/ARsystem may be implemented in the HMD 500, or alternatively may beimplemented at least in part by an external device that iscommunicatively coupled to HMD 500 via a wired or wireless interface.Controller 560 may include one or more of various types of processors,image signal processors (ISPs), graphics processing units (GPUs),coder/decoders (codecs), and/or other components for processing andrendering video and/or images. FIG. 12 further illustrates components ofan HMD and VR/AR system, according to some embodiments.

The HMD 500 may include an eye tracking system that includes, but is notlimited to, one or more eye tracking cameras 540 and an IR light source530. IR light source 530 (e.g., IR LEDs) may be positioned in the HMD500 (e.g., around the eyepieces 520, or elsewhere in the HMD 500) toilluminate the user’s eyes 592 with IR light. At least one eye trackingcamera 540 (e.g., an IR camera, for example a 400×400 pixel count cameraor a 600×600 pixel count camera, that operates at 850 nm or 940 nm, orat some other IR wavelength, and that may be capable of capturing framesat a rate of 60-120 frames per second (FPS)) is located at each side ofthe user 590′s face. In various embodiments, the eye tracking cameras540 may be positioned in the HMD 500 on each side of the user 590′s faceto provide a direct view of the eyes 592, a view of the eyes 592 throughthe eyepieces 520, or a view of the eyes 592 via reflection off hotmirrors or other reflective components. Note that the location and angleof eye tracking camera 540 is given by way of example, and is notintended to be limiting. While FIG. 5 shows a single eye tracking camera540 located on each side of the user 590′s face, in some embodimentsthere may be two or more eye tracking cameras 540 on each side of theuser 590′s face.

To track relative movement of the user’s eyes 592 with respect to theHMD 500, the eye tracking system may also include sensors 544 (e.g.,photosensors or photodiodes, referred to herein as eye motion sensors oreye odometers) placed at one or more positions in the HMD 500 to augmentthe eye tracking cameras 540. The eye odometers 544 may be used as alow-power component to track relative movement of the user’s eyes 592with respect to the HMD 500 in intervals between the processing offrames captured by the cameras 540 to generate 3D models of the user’seyes. The 3D models generated from the images captured by the eyetracking cameras 540 provide absolute gaze information at the frame rateof the cameras, while the data captured by the eye odometers 544 isprocessed in intervals between the processing of the camera frames toprovide relative updates to the 3D models generated from the capturedimages. This may allow the frame rate of the eye tracking cameras 540 tobe reduced, for example from 120 frames per second to 10 frames persecond or less, and may also allow 3D reconstruction based on thecaptured frames to be performed much less often (e.g., 10 times a secondor less), thus significantly reducing power consumption by the eyetracking system. Reducing the frame rate of the eye tracking cameras 540by augmenting eye tracking with the eye odometers 544 may alsosignificantly reduce bandwidth usage and latency of the eye trackingsystem. Examples of different technologies that may be used to implementeye odometers 544 in an eye tracking system are given below in thesection titled Example sensors.

FIG. 6 is a flowchart of an eye tracking method in which sensors areused to track movement of the eyes in intervals between the processingof frames captured by the eye tracking cameras, according to someembodiments. The method of FIG. 6 may, for example, be performed in aVR/AR system as illustrated in FIG. 5 . As indicated at 610, one or moreframes may be captured by the eye tracking cameras. As indicated at 620,a 3D reconstruction method may be performed to generate 3D models of theuser’s eyes using the frame(s) captured by the eye tracking cameras. Forexample, the 3D reconstruction may be performed atinitialization/calibration of the HMD. The 3D models of the eyesindicate the 3D position of the eyes with respect to the eye trackingcameras, which allows the eye tracking algorithms executed by thecontroller to accurately track eye movement with respect to the HMD.

As indicated at 630, after performing an initial 3D reconstruction togenerate 3D models of the eyes, relative movement of the user’s eyeswith respect to the HMD may be tracked using data captured by andcollected from the eye odometers (e.g., photosensors or photodiodes). At640, the eye tracking system may continue to track the user’s eyes usingthe data from the eye odometers until movement of the HMD with respectto the user’s head is detected, a time limit elapses or a confidencethreshold in the eye odometer data is exceeded. If movement of the HMDwith respect to the user’s head is detected, the time limit has elapsedor the confidence threshold has been exceeded, the method returns toelement 610 to capture one or more frames using the eye tracking camerasand perform the 3D reconstruction method 620 to generate new 3D modelsof the user’s eyes using frame(s) captured by the eye tracking cameras.The method then resumes tracking the eyes using the eye odometerinformation at 630.

FIG. 7 illustrates an example VR/AR HMD that implements an eye trackingsystem in which low-resolution frames are captured by the eye trackingcameras and used to track movement of the eyes in intervals between theprocessing of high-resolution frames captured by the eye trackingcameras, according to some embodiments. Note that HMD 700 as illustratedin FIG. 7 is given by way of example, and is not intended to belimiting. In various embodiments, the shape, size, and other features ofan HMD 700 may differ, and the locations, numbers, types, and otherfeatures of the components of an HMD 700 may vary. VR/AR HMD 700 mayinclude, but is not limited to, a display 710 and two eyepieces 720,mounted in a wearable housing or frame. A controller 760 for the VR/ARsystem may be implemented in the HMD 700, or alternatively may beimplemented at least in part by an external device that iscommunicatively coupled to HMD 700 via a wired or wireless interface.Controller 760 may include one or more of various types of processors,image signal processors (ISPs), graphics processing units (GPUs),coder/decoders (codecs), and/or other components for processing andrendering video and/or images. FIG. 12 further illustrates components ofan HMD and VR/AR system, according to some embodiments.

The HMD 700 may include an eye tracking system that includes, but is notlimited to, one or more eye tracking cameras 740 and an IR light source730. IR light source 730 (e.g., IR LEDs) may be positioned in the HMD700 (e.g., around the eyepieces 720, or elsewhere in the HMD 700) toilluminate the user’s eyes 792 with IR light. At least one eye trackingcamera 740 (e.g., an IR camera, for example a 400×400 pixel count cameraor a 600×600 pixel count camera, that operates at 850 nm or 940 nm, orat some other IR wavelength, and that captures frames at a rate of60-120 frames per second (FPS)) is located at each side of the user790′s face. In various embodiments, the eye tracking cameras 740 may bepositioned in the HMD 700 on each side of the user 790′s face to providea direct view of the eyes 792, a view of the eyes 792 through theeyepieces 720, or a view of the eyes 792 via reflection off hot mirrorsor other reflective components. Note that the location and angle of eyetracking camera 740 is given by way of example, and is not intended tobe limiting. While FIG. 7 shows a single eye tracking camera 740 locatedon each side of the user 790′s face, in some embodiments there may betwo or more eye tracking cameras 740 on each side of the user 790′sface.

To track relative movement of the user’s eyes 792 with respect to theHMD 700, instead of including eye odometers 544 in the HMD 500 as shownin FIG. 5 , the eye tracking cameras 740 may capture low-resolutionframes, for example by binning pixels on the camera sensor as shown inFIG. 8A or by capturing horizontal and vertical stripes of pixels on thecamera sensor rather than entire frames as shown in FIG. 8B. In theseembodiments, the cameras 740 may operate in high-resolution mode or inlow-resolution mode. The controller 760 may signal the cameras 740 toswitch to low-resolution mode during or after processing one or morehigh-resolution frames to generate 3D models of the eyes 792, and signalthe cameras 740 to switch to high-resolution mode, for example upondetecting movement of the HMD 700 with respect to the user’s head 790.The low-resolution images may be used to track relative movement of theuser’s eyes 792 with respect to the HMD 700 in intervals between theprocessing of full, high-resolution frames captured by the eye trackingcameras by the controller 760. This may allow high-resolution frames tobe captured much less often, and also 3D reconstruction to be performedmuch less often, thus significantly reducing power consumption by thecameras 740 and controller 760 in the eye tracking system. Reducing thenumber of high-resolution frames that are captured and processed mayalso significantly reduce bandwidth usage and latency of the eyetracking system. The eye tracking cameras 740 themselves may thus beviewed as “eye odometers” when capturing and processing thelow-resolution frames in intervals between the capturing and processingof high-resolution frames.

FIGS. 8A and 8B illustrate non-limiting example low-resolution framesthat may be captured by an eye tracking camera 740 of FIG. 7 , accordingto some embodiments. FIG. 8A shows an example of a frame at 640×640resolution (400k pixels) in high-resolution mode. In low-resolutionmode, the camera 740 bins 32×32 blocks of pixels into one pixel, thusresulting in 20×20 resolution (400 pixels) that may be processed in 2Dto track movement of the user’s eyes in intervals between capturing andprocessing full, high-resolution frames in 3D. FIG. 8B shows an examplein which horizontal and vertical lines of pixels are captured ratherthan an entire frame in low-resolution mode. The captured lines may beprocessed in 2D to track movement of the user’s eyes in intervalsbetween capturing and processing full, high-resolution frames in 3D.

FIG. 9 is a flowchart of an eye tracking method in which low-resolutionframes are captured by the eye tracking cameras and used to trackmovement of the eyes in intervals between the processing ofhigh-resolution frames captured by the eye tracking cameras, accordingto some embodiments. The method of FIG. 9 may, for example, be performedin a VR/AR system as illustrated in FIG. 7 . As indicated at 910, one ormore high-resolution frames may be captured by the eye tracking cameras.As indicated at 920, a 3D reconstruction method may be performed togenerate 3D models of the user’s eyes using the high-resolution frame(s)captured by the eye tracking cameras. For example, the 3D reconstructionmay be performed at initialization/calibration of the HMD. The 3D modelsof the eyes indicate the 3D position of the eyes with respect to the eyetracking cameras, which allows the eye tracking algorithms executed bythe controller to accurately track eye movement with respect to the HMD.

As indicated at 930, after performing an initial 3D reconstruction togenerate 3D models of the eyes, the eye tracking cameras may switch tolow-resolution mode to capture low-resolution frames, for example bybinning pixels on the camera sensor as illustrated in FIG. 8A or bycapturing horizontal and vertical lines of pixels on the camera sensoras illustrated in FIG. 8B. In some embodiments, the controller maysignal the eye tracking cameras to switch to low-resolution mode duringor after processing captured high-resolution frames(s) to generate 3Dmodels of the eyes. As indicated at 940, movement of the user’s eyeswith respect to the HMD may be tracked using the low-resolution imagescaptured in low-resolution mode. At 950, the eye tracking system maycontinue to capture low-resolution images and track the user’s eyes withrespect to the HMD using the low-resolution information until movementof the HMD with respect to the user’s head is detected, a time limitelapses, or a confidence threshold is exceeded. At 950, if movement ofthe HMD with respect to the user’s head is detected, the time limit haselapsed, or the confidence threshold has been exceeded, the methodreturns to element 910 to capture one or more high-resolution framesusing the eye tracking cameras and perform the 3D reconstruction method920 to generate new 3D models of the user’s eyes using frame(s) capturedby the eye tracking cameras. In some embodiments, the controller maysignal the eye tracking cameras to switch to high-resolution mode upondetecting movement of the HMD with respect to the user’s head,determining that the time limit has elapsed, or determining that theconfidence threshold has been exceeded. The eye tracking system thenresumes capturing low-resolution images at 930 and tracking the eyesusing the low-resolution images at 940.

The head odometer methods and eye odometer methods described above inreference to FIGS. 2 through 9 may be used in combination. Embodimentsof HMDs that include both head odometers and eye odometers are describedin reference to FIGS. 10 and 11 .

FIG. 10 illustrates an example VR/AR HMD that implements an eye trackingsystem that includes head odometers that detect movement of the HMD withrespect to the user’s eyes and eye odometers that track movement of theeyes in intervals between the processing of frames captured by the eyetracking cameras, according to some embodiments. Note that HMD 1000 asillustrated in FIG. 10 is given by way of example, and is not intendedto be limiting. In various embodiments, the shape, size, and otherfeatures of an HMD 1000 may differ, and the locations, numbers, types,and other features of the components of an HMD 1000 may vary. VR/AR HMD1000 may include, but is not limited to, a display 1010 and twoeyepieces 1020, mounted in a wearable housing or frame. A controller1060 for the VR/AR system may be implemented in the HMD 1000, oralternatively may be implemented at least in part by an external devicethat is communicatively coupled to HMD 1000 via a wired or wirelessinterface. Controller 1060 may include one or more of various types ofprocessors, image signal processors (ISPs), graphics processing units(GPUs), coder/decoders (codecs), and/or other components for processingand rendering video and/or images. FIG. 12 further illustratescomponents of an HMD and VR/AR system, according to some embodiments.

The HMD 1000 may include an eye tracking system that includes, but isnot limited to, one or more eye tracking cameras 1040 and an IR lightsource 1030. IR light source 1030 (e.g., IR LEDs) may be positioned inthe HMD 1000 (e.g., around the eyepieces 1020, or elsewhere in the HMD1000) to illuminate the user’s eyes 1092 with IR light. At least one eyetracking camera 1040 (e.g., an IR camera, for example a 400×400 pixelcount camera or a 600×600 pixel count camera, that operates at 850 nm or940 nm, or at some other IR wavelength, and that captures frames at arate of 60-120 frames per second (FPS)) is located at each side of theuser 1090′s face. In various embodiments, the eye tracking cameras 1040may be positioned in the HMD 1000 on each side of the user 1090′s faceto provide a direct view of the eyes 1092, a view of the eyes 1092through the eyepieces 1020, or a view of the eyes 1092 via reflectionoff hot mirrors or other reflective components. Note that the locationand angle of eye tracking camera 1040 is given by way of example, and isnot intended to be limiting. While FIG. 10 shows a single eye trackingcamera 1040 located on each side of the user 1090′s face, in someembodiments there may be two or more eye tracking cameras 1040 on eachside of the user 1090′s face.

To track relative movement of the HMD 1000 with respect to the user’seyes, the eye tracking system may also include head odometers 1042placed at one or more positions in or on the device, for example at ornear the user’s ears (1042A) to primarily track pitch and at or near thebridge of the nose (1042B) to primarily track y movement. Signals fromthe head odometers 1042A and 1042B may be processed by controller 1060to detect movement of the HMD 1000 with respect to the user’s eyes. Thismay allow 3D reconstruction to be performed by controller 1060 only whenmovement of the HMD 1000 with respect to the user’s eyes has beendetected, thus significantly reducing power consumption by the eyetracking system. In some embodiments, instead of performing 3Dreconstruction when movement of the HMD 1000 with respect to the user’seyes 1092 has been detected, magnitude and direction of the detectedmotion may be determined by controller 1060, and 3D models of the user’seyes 1092 previously generated by the 3D reconstruction method may beadjusted by controller 1060 according to the magnitude and direction ofthe detected motion of the HMD 1000.

To track relative movement of the user’s eyes 1092 with respect to theHMD 1000, the eye tracking system may also include eye odometers 1044placed at one or more positions in the HMD 1000 to augment the eyetracking cameras 1040. The eye odometers 1044 may be used as a low-powercomponent to track relative movement of the user’s eyes 1092 withrespect to the HMD 500 in intervals between the processing of framescaptured by the cameras 1040. The 3D models generated by controller 1060from the images captured by the eye tracking cameras 1040 provideabsolute gaze information, while the data captured by the eye odometers1044 in intervals between the processing of camera frames is processedby controller 1060 to provide a relative update to the previously knownand trusted 3D models. This may allow the frame rate of the eye trackingcameras 1040 to be reduced, for example from 120 frames per second to 10frames per second or less, and may also allow 3D reconstruction to beperformed by controller 1060 much less often, thus significantlyreducing power consumption by the eye tracking system. Augmenting eyetracking with the eye odometers 1044 may also significantly reducebandwidth usage and latency of the eye tracking system. Examples ofdifferent technologies that may be used to implement eye odometers 1044in an eye tracking system are given below in the section titled Examplesensors.

As an alternative, in some embodiments, instead of using eye odometers1044 in the HMD 1000 to track eye movement in intervals between frames,the eye tracking cameras 1040 may capture low-resolution frames inintervals between capturing high-resolution frames, for example bybinning pixels on the camera sensor or by capturing horizontal andvertical stripes or lines of pixels on the camera sensor rather thanentire frames. This low-resolution information may be processed bycontroller 1060 to track relative movement of the user’s eyes 1092 withrespect to the HMD 1000 in intervals between full, high-resolutionframes captured by the eye tracking cameras 1040 and processed by thecontroller 1060.

FIG. 11 is a flowchart of an eye tracking method in which head odometersare used detect movement of the HMD with respect to the user’s eyes andeye odometers are used to track movement of the eyes in intervalsbetween the processing of frames captured by the eye tracking cameras,according to some embodiments. The method of FIG. 11 may, for example,be performed in a VR/AR system as illustrated in FIG. 10 . As indicatedat 1110, one or more frames may be captured by the eye tracking cameras.As indicated at 1120, a 3D reconstruction method may be performed togenerate 3D models of the user’s eyes using the frame(s) captured by theeye tracking cameras. For example, the 3D reconstruction may beperformed at initialization/calibration of the HMD. The 3D models of theeyes indicate the 3D position of the eyes with respect to the eyetracking cameras, which allows the eye tracking algorithms executed bythe controller to accurately track eye movement with respect to the HMD.

As indicated at 1130, after performing an initial 3D reconstruction togenerate 3D models of the eyes, movement of the user’s eyes with respectto the HMD may be tracked in 2D space using data captured by andcollected from the eye odometers on the HMD. As indicated at 1140,relative movement of the HMD with respect to the user’s head may betracked using head odometers on the HMD.

At 1150, if the controller detects movement of the HMD with respect tothe user’s eyes from the data received from the head odometers, themethod returns to element 1110 to again perform the 3D reconstructionmethod to generate new 3D models of the user’s eyes using frame(s)captured by the eye tracking cameras. Alternatively, magnitude anddirection of the movement may be determined from the head odometersignals. This information may then be used by the controller to adjustthe 3D models, which allows the eye tracking algorithms executed by thecontroller to continue to accurately track eye movement with respect tothe HMD in 2D space without requiring 3D reconstruction to be performed.

At 1150, if movement of the HMD with respect to the user’s head is notdetected, and if a time limit has not elapsed at 1160, then the eyetracking method may continue at 1130 to track movement of the user’seyes with respect to the HMD using data from the eye odometers and at1140 to track movement of the HMD with respect to the HMD using datafrom the head odometers. At 1160, if the time limit has elapsed, and ifa confidence threshold for the eye tracking data has not been exceededat 1170, then the eye tracking method may continue at 1130 to trackmovement of the user’s eyes with respect to the HMD using data from theeye odometers and at 1140 to track movement of the HMD with respect tothe HMD using data from the head odometers. At 1170, if the confidencethreshold has been exceeded, the method returns to element 1110 to againperform the 3D reconstruction method to generate new 3D models of theuser’s eyes using frame(s) captured by the eye tracking cameras.

Embodiments of an eye tracking system for an HMD that include both headodometers and eye odometers as illustrated in FIGS. 10 and 11 may, forexample, further reduce the frequency at which 3D reconstruction isperformed, and may also reduce the frequency at which 2D imageprocessing of frames captured by the eye tracking cameras is performedto further reduce power consumption of the eye tracking system.

Example Sensors

The following provides non-limiting examples of different technologiesthat may be used to implement head motion and eye motion sensors in aneye tracking system as illustrated in FIGS. 2 through 13 . Thesetechnologies can be broken into three broad categories: driftcompensation (1D) technologies, odometry (2D) technologies, and absolutetracking (5D) technologies.

Drift compensation technologies may work by contact, and may includeodometry sensors in the HMD, for example in a nose pad of the HMD thatmake contact on the bridge of the user’s nose and at the temple area ofthe HMD that make contact near the user’s ears. The odometry sensors maymeasure force. Instead or in addition, the sensors may be capacitivesensors. As an alternative, accelerometers with a smart baseline may beused.

Odometry technologies may be broken into five categories: electric,optical, acoustic, radar, and pressure technologies. Electric odometrytechnologies may include, but are not limited to, EOG/EMG, ECG, andcapacity sensor (waveguide or triangulation) technologies. Opticalodometry technologies may include, but are not limited to, photosensor,photodiode array, proximity sensor, and scanner (TOF, intensity, orphase) technologies. Acoustic odometry technologies may include, but arenot limited to, sonar (continuous wave, TOF, or resonance) technologies.Radar odometry technologies may include, but are not limited to,continuous wave or TOF radar technologies. Pressure odometrytechnologies may include, but are not limited to, an IMU (inertialmeasurement unit) or other sensor that senses pressure change from eyemovement.

Absolute tracking technologies may be broken into two categories:optical and sonar technologies. Optical absolute tracking technologiesmay include, but are not limited to, single pixel TOF sensor technologyor various camera technologies. The camera technologies may include, butare not limited to, DVS, CIS (RGB + IR), compressed sensing,skipping/binning (e.g., slit camera), and thermal imaging (short,medium, long, pyro-electric, or thermo-electric) technologies. Sonarabsolute tracking technologies may include, but are not limited to,transceiver array, line of sight, and non-line of sight sonartechnologies.

FIG. 12 is a block diagram illustrating an example VR/AR system thatincludes components of an eye tracking system as illustrated in FIGS. 2through 11 , according to some embodiments. In some embodiments, a VR/ARsystem may include an HMD 2000 such as a headset, helmet, goggles, orglasses. HMD 2000 may implement any of various types of virtual realityprojector technologies. For example, the HMD 2000 may include a VRprojection system that includes a projector 2020 that displays framesincluding left and right images on screens or displays 2022A and 2022Bthat are viewed by a user through eyepieces 2220A and 2220B. The VRprojection system may, for example, be a DLP (digital light processing),LCD (liquid crystal display), or LCoS (liquid crystal on silicon)technology projection system. To create a three-dimensional (3D) effectin a 3D virtual view, objects at different depths or distances in thetwo images may be shifted left or right as a function of thetriangulation of distance, with nearer objects shifted more than moredistant objects. Note that other types of projection systems may be usedin some embodiments.

In some embodiments, HMD 2000 may include a controller 2030 configuredto implement functionality of the VR/AR system and to generate frames(each frame including a left and right image) that are displayed by theprojector 2020. In some embodiments, HMD 2000 may also include a memory2032 configured to store software (code 2034) of the VR/AR system thatis executable by the controller 2030, as well as data 2038 that may beused by the VR/AR system when executing on the controller 2030. In someembodiments, HMD 2000 may also include one or more interfaces (e.g., aBluetooth technology interface, USB interface, etc.) configured tocommunicate with an external device 2100 via a wired or wirelessconnection. In some embodiments, at least a part of the functionalitydescribed for the controller 2030 may be implemented by the externaldevice 2100. External device 2100 may be or may include any type ofcomputing system or computing device, such as a desktop computer,notebook or laptop computer, pad or tablet device, smartphone, hand-heldcomputing device, game controller, game system, and so on.

In various embodiments, controller 2030 may be a uniprocessor systemincluding one processor, or a multiprocessor system including severalprocessors (e.g., two, four, eight, or another suitable number).Controller 2030 may include central processing units (CPUs) configuredto implement any suitable instruction set architecture, and may beconfigured to execute instructions defined in that instruction setarchitecture. For example, in various embodiments controller 2030 mayinclude general-purpose or embedded processors implementing any of avariety of instruction set architectures (ISAs), such as the x86,PowerPC, SPARC, RISC, or MIPS ISAs, or any other suitable ISA. Inmultiprocessor systems, each of the processors may commonly, but notnecessarily, implement the same ISA. Controller 2030 may employ anymicroarchitecture, including scalar, superscalar, pipelined,superpipelined, out of order, in order, speculative, non-speculative,etc., or combinations thereof. Controller 2030 may include circuitry toimplement microcoding techniques. Controller 2030 may include one ormore processing cores each configured to execute instructions.Controller 2030 may include one or more levels of caches, which mayemploy any size and any configuration (set associative, direct mapped,etc.). In some embodiments, controller 2030 may include at least onegraphics processing unit (GPU), which may include any suitable graphicsprocessing circuitry. Generally, a GPU may be configured to renderobjects to be displayed into a frame buffer (e.g., one that includespixel data for an entire frame). A GPU may include one or more graphicsprocessors that may execute graphics software to perform a part or allof the graphics operation, or hardware acceleration of certain graphicsoperations. In some embodiments, controller 2030 may include one or moreother components for processing and rendering video and/or images, forexample image signal processors (ISPs), coder/decoders (codecs), etc.

Memory 2032 may include any type of memory, such as dynamic randomaccess memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR,DDR2, DDR3, etc.) SDRAM (including mobile versions of the SDRAMs such asmDDR3, etc., or low power versions of the SDRAMs such as LPDDR2, etc.),RAMBUS DRAM (RDRAM), static RAM (SRAM), etc. In some embodiments, one ormore memory devices may be coupled onto a circuit board to form memorymodules such as single inline memory modules (SIMMs), dual inline memorymodules (DIMMs), etc. Alternatively, the devices may be mounted with anintegrated circuit implementing system in a chip-on-chip configuration,a package-on-package configuration, or a multi-chip moduleconfiguration.

In some embodiments, the HMD 2000 may include one or more sensors 2050that collect information about the user’s environment (video, depthinformation, lighting information, etc.). The sensors 2050 may providethe information to the controller 2030 of the VR/AR system. In someembodiments, sensors 2050 may include, but are not limited to, visiblelight cameras (e.g., video cameras).

As shown in FIGS. 1, 2, 5, 7 and 10 , HMD 2000 may be positioned on theuser’s head such that the displays 2022A and 2022B and eyepieces 2220Aand 2220B are disposed in front of the user’s eyes 2292A and 2292B. IRlight sources 2230A and 2230B (e.g., IR LEDs) may be positioned in theHMD 2000 (e.g., around the eyepieces 2220A and 2220B, or elsewhere inthe HMD 2000) to illuminate the user’s eyes 2292A and 2292B with IRlight. Eye tracking cameras 2240A and 2240B (e.g., IR cameras, forexample 400×400 pixel count cameras or 600×600 pixel count cameras thatoperate at 850 nm or 940 nm, or at some other IR wavelength, and thatcapture frames at a rate of 60-120 frames per second (FPS)) are locatedat each side of the user’s face. In various embodiments, the eyetracking cameras 2240 may be positioned in the HMD 2000 to provide adirect view of the eyes 2292, a view of the eyes 2292 through theeyepieces 2220, or a view of the eyes 2292 via reflection off hotmirrors or other reflective components. Note that the location and angleof eye tracking cameras 2240A and 2240B is given by way of example, andis not intended to be limiting. In some embodiments, there may be asingle eye tracking camera 2240 located on each side of the user’s face.In some embodiments there may be two or more eye tracking cameras 2240on each side of the user’s face. For example, in some embodiments, awide-angle camera 2240 and a narrower-angle camera 2240 may be used oneach side of the user’s face. A portion of IR light emitted by lightsources 2230A and 2230B reflects off the user’s eyes 2292A and 2292B isreceived at respective eye tracking cameras 2240A and 2240B, and iscaptured by the eye tracking cameras 2240A and 2240B to image the user’seyes 2292A and 2292B. Eye tracking information captured by the cameras2240A and 2240B may be provided to the controller 2030. The controller2030 may analyze the eye tracking information (e.g., images of theuser’s eyes 2292A and 2292B) to determine eye position and movementand/or other features of the eyes 2292A and 2292B. In some embodiments,to accurately determine the location of the user’s eyes 2292A and 2292Bwith respect to the eye tracking cameras 2240A and 2240B, the controller2030 may perform a 3D reconstruction using images captured by the eyetracking cameras 2240A and 2240B to generate 3D models of the user’seyes 2292A and 2292B. The 3D models of the eyes 2292A and 2292B indicatethe 3D position of the eyes 2292A and 2292B with respect to the eyetracking cameras 2240A and 2240, which allows the eye trackingalgorithms executed by the controller to accurately track eye movement.

In some embodiments, HMD 2000 may include sensors 2242 (referred toherein as head motion sensors or head odometers) placed at one or morepositions in or on the device, for example at or near the user’s ears(2242A) to primarily track pitch and at or near the bridge of the nose(2242B) to primarily track y movement. Signals from the head odometers2242A and 2242B may be used to detect movement of the HMD 2000 withrespect to the user’s eyes 2292A and 2292B. This may allow 3Dreconstruction to be performed only when movement of the HMD 2000 withrespect to the user’s eyes 2292A and 2292B has been detected, thussignificantly reducing power consumption by the eye tracking system.When no movement of the HMD 2000 is detected, 2D image processing offrames captured by the eye tracking cameras 2240A and 2240B (which ismuch less expensive computationally than 3D image processing) may beperformed to track the user’s eyes 2292A and 2292B. In some embodiments,a 3D reconstruction may be performed periodically (e.g., once a second,or once every N frames) to prevent error/drift accumulation even ifmovement of the HMD 2000 has not been detected. In some embodiments,instead of performing 3D reconstruction when movement of the HMD 2000with respect to the user’s eyes 2292A and 2292B has been detected,magnitude and direction of the detected motion may be determined, and 3Dmodels of the user’s eyes 2292A and 2292B that were previously generatedby the 3D reconstruction method may be adjusted according to themagnitude and direction of the detected motion of the HMD 2000. In theseembodiments, a 3D reconstruction may be performed periodically (e.g.,once a second, or once every N frames) to prevent error/driftaccumulation.

In some embodiments, HMD 2000 may include sensors 2244 (e.g.,photosensors or photodiodes, referred to herein as eye motion sensors oreye odometers) placed at one or more positions in the device to augmentthe eye tracking cameras 2240A and 2240B. The eye odometers 2244A and2244B may be used as a low-power component to track relative movement ofthe user’s eyes 2292A and 2292B with respect to the cameras 2240 inintervals between the processing of frames captured by the cameras 2240Aand 2240B. This may allow the frame rate of the eye tracking cameras2240A and 2240B to be reduced, for example from 120 frames per second to10 frames per second, and may also allow 3D reconstruction to beperformed much less often, thus significantly reducing power consumptionby the eye tracking system. Reducing the frame rate of the eye trackingcameras 2240A and 2240B by augmenting eye tracking with the eyeodometers 2244A and 2244B also significantly reduces bandwidth usage ofthe eye tracking system.

In some embodiments, instead of using sensors 2244A and 2244B to augmentthe eye tracking cameras 2240A and 2240B, the eye tracking cameras 2240Aand 2240B may capture lower-resolution frames, for example by binningpixels or by capturing horizontal and vertical stripes of pixels, ratherthan entire high-resolution frames. The low-resolution frames capturedby cameras 2240A and 2240B may be used to track relative movement of theuser’s eyes 2292A and 2292B with respect to the cameras 2240A and 2240Bin intervals between the processing of full, high-resolution framescaptured by the eye tracking cameras 2240A and 2240B. This may allow 3Dreconstruction to be performed much less often, thus significantlyreducing power consumption by the eye tracking system. In theseembodiments, the eye tracking cameras 2240A and 2240B themselves may beviewed as “eye odometers” when capturing and processing thelower-resolution frames.

The head odometer methods and eye odometer methods described above maybe used alone or in combination. Embodiments of an HMD 2000 may includehead odometers 2242 to reduce power consumption of the eye trackingsystem. Embodiments of an HMD 2000 may include eye odometers 2244(either implemented by sensors or by the eye tracking cameras) to reducepower consumption and bandwidth usage of the eye tracking system. Inaddition, embodiments of an HMD 2000 may include both head odometers2242 and eye odometers 2244. Embodiments that include both headodometers 2242 and eye odometers 2244 may, for example, further reducethe frequency at which 3D reconstruction is performed, and may alsoreduce the frequency at which 2D image processing of frames captured bythe eye tracking cameras 2240 is performed to further reduce powerconsumption of the eye tracking system.

The eye tracking information obtained and analyzed by the controller2030 may be used by the controller in performing various VR or AR systemfunctions. For example, the point of gaze on the displays 2022A and2022B may be estimated from images captured by the eye tracking cameras2240A and 2240B; the estimated point of gaze may, for example, enablegaze-based interaction with content shown on the displays 2022A and2022B. Other applications of the eye tracking information may include,but are not limited to, creation of eye image animations used foravatars in a VR or AR environment. As another example, in someembodiments, the information obtained from the eye tracking cameras2240A and 2240B may be used to adjust the rendering of images to beprojected, and/or to adjust the projection of the images by theprojector 2020 of the HMD 2000, based on the direction and angle atwhich the user’s eyes are looking. As another example, in someembodiments, brightness of the projected images may be modulated basedon the user’s pupil dilation as determined by the eye tracking system.

In some embodiments, the HMD 2000 may be configured to render anddisplay frames to provide an augmented or mixed reality (AR) view forthe user at least in part according to sensor 2050 inputs. The AR viewmay include renderings of the user’s environment, including renderingsof real objects in the user’s environment, based on video captured byone or more video cameras that capture high-quality, high-resolutionvideo of the user’s environment for display. The AR view may alsoinclude virtual content (e.g., virtual objects, virtual tags for realobjects, avatars of the user, etc.) generated by VR/AR system andcomposited with the projected view of the user’s real environment.

Embodiments of the HMD 2000 as illustrated in FIG. 12 may also be usedin virtual reality (VR) applications to provide VR views to the user. Inthese embodiments, the controller 2030 of the HMD 2000 may render orobtain virtual reality (VR) frames that include virtual content, and therendered frames may be provided to the projector 2020 of the HMD 2000for display to displays 2022A and 2022B.

While embodiments are generally described in FIGS. 2 through 12 withrespect to an example VR/AR system and HMD that includes near-eyedisplay panels that the user views through eyepieces, embodiments of thelow-power eye system methods and apparatus may be implemented in othertypes of VR/AR systems, and more generally in any system in which eyetracking is performed using eye tracking cameras. FIG. 13 illustrates anexample alternative VR/AR device 3000 that includes an eye trackingsystem with head odometers 3042 that detect movement of the device 3000with respect to the user’s eyes 3092 and eye odometers 3044 that trackmovement of the eyes 3092 in intervals between the processing of framescaptured by the eye tracking cameras 3040, according to someembodiments. Note that device 3000 as illustrated in FIG. 13 is given byway of example, and is not intended to be limiting. In variousembodiments, the shape, size, and other features of the device 3000 maydiffer, and the locations, numbers, types, and other features of thecomponents of the device 3000 may vary. In this device 3000, rather thanhaving display panels that the user views through eyepieces, the device3000 includes lenses 3010 that allow light from the outside environmentto pass, and that are also configured to reflect light emitted by aprojector/controller 3080 representing virtual content towards theuser’s eyes 3092, for example using a holographic film. Device 3000 mayinclude an eye tracking system that, in addition to eye tracking cameras3040, includes head odometers 3042 that detect movement of the device3000 with respect to the user’s eyes 3092 as described herein, eyeodometers 3044 that track movement of the eyes 3092 in intervals betweenthe processing of frames captured by the eye tracking cameras 3040 asdescribed herein, or both head odometers 3042 and eye odometers 3044 asdescribed herein.

The methods described herein may be implemented in software, hardware,or a combination thereof, in different embodiments. In addition, theorder of the blocks of the methods may be changed, and various elementsmay be added, reordered, combined, omitted, modified, etc. Variousmodifications and changes may be made as would be obvious to a personskilled in the art having the benefit of this disclosure. The variousembodiments described herein are meant to be illustrative and notlimiting. Many variations, modifications, additions, and improvementsare possible. Accordingly, plural instances may be provided forcomponents described herein as a single instance. Boundaries betweenvarious components, operations and data stores are somewhat arbitrary,and particular operations are illustrated in the context of specificillustrative configurations. Other allocations of functionality areenvisioned and may fall within the scope of claims that follow. Finally,structures and functionality presented as discrete components in theexample configurations may be implemented as a combined structure orcomponent. These and other variations, modifications, additions, andimprovements may fall within the scope of embodiments as defined in theclaims that follow.

What is claimed is: 1-20. (canceled)
 21. A system, comprising: ahead-mounted device (HMD), comprising a camera configured to captureimages of an eye; a controller comprising one or more processorsconfigured to: perform a 3D reconstruction to generate a 3D model of theeye using the captured images; track movement of the eye with respect tothe HMD using data captured by one or more eye motion sensors; trackposition of the HMD with respect to the eye using data collected fromone or more head motion sensors; and upon detecting a shift of theposition of the HMD with respect to the eye, update or regenerate the 3Dmodel of the eye.
 22. The system as recited in claim 21, wherein thecontroller is further configured to update or regenerate the 3D model ofthe eye upon detecting that a time limit has elapsed and a confidencethreshold for eye tracking data has been exceeded.
 23. The system asrecited in claim 21, wherein the one or more eye motion sensors includeone or more of photosensors and photodiode arrays.
 24. The system asrecited in claim 21, wherein the one or more eye motion sensors includethe camera in low-resolution mode, wherein the controller is configuredto process one or more low-resolution frames captured by the camera inthe low-resolution mode using two-dimensional (2D) image processingtechniques to track movement of the eye in 2D space with respect to theHMD in intervals between high-resolution frames captured by the camera.25. The system as recited in claim 24, wherein the low-resolution framesare captured by binning blocks of pixels on a camera sensor.
 26. Thesystem as recited in claim 24, wherein the low-resolution frames arecaptured by capturing horizontal and vertical lines of pixels on acamera sensor.
 27. The system as recited in claim 21, wherein the one ormore head motion sensors include at least one sensor located in the HMDat or near the bridge of a nose and at least one sensor located in theHMD at or near a temple.
 28. The system as recited in claim 21, wherein,to update the 3D model of the eye, the controller is configured toadjust an initial 3D model of the eye based on magnitude and directionof movement of the HMD with respect to the eye determined from the datacollected from the one or more head motion sensors.
 29. The system asrecited in claim 21, wherein the controller is a component of the HMD.30. The system as recited in claim 21, wherein the HMD furthercomprises: a display screen configured to display frames containingvisual content, wherein the controller is further configured to renderthe frames containing the visual content for display by the displayscreen; and one or more light sources configured to emit light towardsthe eye, wherein the camera captures a portion of the light reflectedoff the eye.
 31. The system as recited in claim 29, wherein the HMDfurther comprises an optical lens located between the display screen andthe eye.
 32. A method, comprising: performing, by one or moreprocessors: generating an initial three-dimensional (3D) model of an eyebased on one or more images of the eye captured by a camera of ahead-mounted device (HMD); tracking movement of the eye with respect tothe HMD using data captured by one or more eye motion sensors; trackingposition of the HMD with respect to the eye using data collected fromone or more head motion sensors; and upon detecting a shift of theposition of the HMD with respect to the eye, updating or regeneratingthe 3D model of the eye.
 33. The method as recited in claim 32, furthercomprising updating or regenerating the 3D model of the eye upondetecting that a time limit has elapsed and a confidence threshold foreye tracking data has been exceeded.
 34. The method as recited in claim32, wherein the one or more eye motion sensors include one or more ofphotosensors and photodiode arrays.
 35. The method as recited in claim32, wherein the one or more eye motion sensors include the camera inlow-resolution mode, the method further comprising processing one ormore low-resolution frames captured by the camera in the low-resolutionmode using two-dimensional (2D) image processing techniques to trackmovement of the eye in 2D space with respect to the HMD in intervalsbetween high-resolution frames captured by the camera.
 36. The method asrecited in claim 35, wherein the low-resolution frames are captured bybinning blocks of pixels on a camera sensor.
 37. The method as recitedin claim 35, wherein the low-resolution frames are captured by capturinghorizontal and vertical lines of pixels on a camera sensor.
 38. Themethod as recited in claim 32, wherein the one or more head motionsensors include at least one sensor located in the HMD at or near thebridge of a nose and at least one sensor located in the HMD at or near atemple.
 39. The method as recited in claim 32, wherein updating the 3Dmodel of the eye comprises adjusting the initial 3D model of the eyebased on magnitude and direction of movement of the HMD with respect tothe eye determined from the data collected from the one or more headmotion sensors.
 40. The method as recited in claim 32, furthercomprising rendering frames containing visual content for display by adisplay screen of the HMD.